Experiment with asynchrony in multimodal speech communication

The purpose of this study was to examine the delay effects in audiovisual speech perception for natural and synthetic faces. The main focus was on the SYNFACE project, the development of a telephone communication aid for hearing impaired persons. In the experiments, the consequence of temporal displacement of the audio in relation to the visual channel was investigated. The audio channel was natural speech with a vocoder-like distortion to simulate hearing loss. Twelve different experimental conditions were presented to the subjects in two separate sessions. The natural face was tested for audio-leading (negative numbers) as well as audio-lagging (positive numbers) stimuli, whereas the synthetic face was tested only for audio-leading stimuli. Asynchronies examined were 50, 175 and 300 ms. In addition, two reference conditions were examined: synchrony and audio-only. Tests of ANOVA including both faces revealed that neither -300 ms nor 175 ms were significantly better than the audio-only condition, which implies that the final SYNFACE product would not be beneficial for delays of this magnitude. The -50 ms condition, however, did not show significantly lower intelligibility scores than the synchronous condition. Unfortunately, the delay measured in the present SYNFACE prototype is greater than this. It would, therefore, be interesting to investigate asynchronies between -175 ms and -50 ms to see exactly where the intelligibility drops. ANOVA further showed that the effect of the type of face was non-significant, indicating that the quality of the synthetic face is close to that of a natural face. Experiment with asynchrony in multimodal speech communication v The tolerance for audio-lagging delays is larger than for the audio-leading delays, which is verified by a significant decrease in performance as late as at +300 ms (the corresponding audio-leading delay is -175 ms). Even a gain in intelligibility was found for the +50 ms condition compared to synchrony. However, this gain is not significant, and statistical analysis showed that delays within the interval [-50, +175] only have a small effect on the spoken message for the natural face.